The 80% nobody tells you about
Here is the truth that every AI course glosses over: in real professional AI projects, around 70 to 80% of all the work is preparing the data, not building or tuning models. Loading a CSV and training a model takes minutes. Dealing with a dataset where 30% of the age column is missing, rows are duplicated, salaries are stored as text, and several entries are physically impossible? That takes days.
This is not a complaint. Understanding your data deeply before modelling is one of the most important skills in all of AI. A model trained on dirty data will give you dirty predictions, often confidently and invisibly. Garbage in, garbage out is not just a saying. It is the most common cause of failed AI projects.
"Data preparation accounts for about 80% of the work of data scientists."
Kaggle State of Data Science surveyThe EDA checklist: run this on every new dataset
Exploratory Data Analysis (EDA) is the process of getting to know a dataset before you do anything with it. Every experienced data scientist runs through a mental checklist when they first open a new dataset. Here it is, with the exact Pandas commands.
Handling missing values
Missing data is the most common data quality problem. It appears in almost every real dataset. The question is not whether you will encounter it, but how you will handle it. You have several options, and the right choice depends on the situation.
# Find how many nulls in each column df.isnull().sum() # Percentage of nulls per column (df.isnull().sum() / len(df)) * 100 ## OPTION 1: Drop rows with ANY null value df_clean = df.dropna() ## OPTION 2: Drop only if a SPECIFIC column is null df_clean = df.dropna(subset=['Age', 'Fare']) ## OPTION 3: Fill nulls with the column mean (imputation) df['Age'] = df['Age'].fillna(df['Age'].mean()) ## OPTION 4: Fill with the most common value (for categories) df['Embarked'] = df['Embarked'].fillna(df['Embarked'].mode()[0])
Drop rows if very few are affected and the remaining dataset is still large enough. Fill (impute) when dropping would remove too much data. Never just fill all nulls with zero. A zero age means something completely different from a missing age. The choice you make here matters.
Before and after: what cleaning actually looks like
Embarked: S, C, NaN, Q, S
Name: "Braund", "Braund", ← duplicate
Fare: 7.25, -999, 71.28 ← outlier
Cabin: NaN (77% missing)
Embarked: S, C, S, Q, S
Name: "Braund" ← duplicate removed
Fare: 7.25, dropped, 71.28
Cabin: column dropped (too sparse)
Finding and handling outliers
An outlier is a value so extreme it could distort your model. But before you remove an outlier, you need to understand it. Is it a data entry error? Is it a legitimate extreme case? Removing a billionaire from an income dataset might be statistically helpful but analytically dishonest if you are trying to understand income distribution.
# IQR (interquartile range) method Q1 = df['Fare'].quantile(0.25) Q3 = df['Fare'].quantile(0.75) IQR = Q3 - Q1 lower = Q1 - 1.5 * IQR upper = Q3 + 1.5 * IQR # Find the outliers outliers = df[(df['Fare'] < lower) | (df['Fare'] > upper)] print(f"Outliers found: {len(outliers)}") # Remove outliers (only if you have a good reason) df_clean = df[(df['Fare'] >= lower) & (df['Fare'] <= upper)] # Visualise with a box plot — outliers appear as dots beyond the whiskers import matplotlib.pyplot as plt df.boxplot(column=['Fare', 'Age']) plt.show()
Encoding categorical data
Machine learning models cannot work with text categories like "male/female" or "S/C/Q". You need to convert them to numbers. There are two common methods.
## METHOD 1: Label encoding (0, 1, 2...) ## Good for ordinal data (small, medium, large) df['Sex_encoded'] = df['Sex'].map({'male': 0, 'female': 1}) ## METHOD 2: One-hot encoding (dummy variables) ## Good for nominal data — avoids false ordering df = pd.get_dummies(df, columns=['Embarked']) # Creates new columns: Embarked_C, Embarked_Q, Embarked_S # Each is 0 or 1 — no false ordering implied
If you encode "red=1, green=2, blue=3", the model might think green is twice as important as red and blue is three times as important, which is complete nonsense for colours. One-hot encoding gives each category its own column of 0s and 1s so the model treats them equally with no implied ordering.
A complete cleaning pipeline
import pandas as pd import matplotlib.pyplot as plt # 1. Load url = "https://raw.githubusercontent.com/datasciencedojo/datasets/master/titanic.csv" df = pd.read_csv(url) # 2. Explore print(df.shape) print(df.isnull().sum()) df.describe() # 3. Drop columns with too many missing values df = df.drop(columns=['Cabin', 'Ticket', 'Name', 'PassengerId']) # 4. Fill missing values df['Age'] = df['Age'].fillna(df['Age'].median()) df['Embarked'] = df['Embarked'].fillna('S') # 5. Encode categorical columns df['Sex'] = df['Sex'].map({'male': 0, 'female': 1}) df = pd.get_dummies(df, columns=['Embarked']) # 6. Remove duplicates df = df.drop_duplicates() # 7. Confirm the result print("Clean dataset shape:", df.shape) print("Remaining nulls:", df.isnull().sum().sum()) df.head()
The output of this pipeline is a clean, fully numeric DataFrame with no missing values and no duplicate rows. It is now ready to be fed into a machine learning model. That is exactly what you will do in Phase 3. Every column is a number. Every row is a valid observation. This is what a model needs to learn.
